Skip to content

drivers: bound MSI-X vector use so many-queue guests still boot - #1465

Open
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/msix-vector-budget
Open

drivers: bound MSI-X vector use so many-queue guests still boot#1465
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/msix-vector-budget

Conversation

@gburd

@gburd gburd commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The x86-64 IDT exposes only 224 usable interrupt vectors (32..255), shared by every device. A guest with many vCPUs and several multiqueue virtio devices can request one MSI-X vector per queue and exhaust the pool during boot. When that happened two things went wrong:

  • register_interrupt_handler() called abort() when no IDT vector was free, crashing the boot instead of letting the device degrade.
  • request_vectors() pushed a msix_vector whose allocation had failed, and ~msix_vector() then unregistered vector 0.

This bounds vector use so the guest boots either way:

  • register_interrupt_handler() returns 0 (an impossible real vector, since 0..31 are CPU exceptions) as an "exhausted" sentinel instead of aborting. msix_vector's destructor skips the unregister when its vector is 0, and request_vectors() stops at the first failed allocation and returns the short vector it did get. The existing easy_register() already treats a short result as failure, so a driver falls back to fewer queues rather than crashing.
  • Adds virtio_driver::reserve_msix_vectors(), a process-wide soft budget a multiqueue driver consults to cap how many queues it arms with a distinct interrupt, leaving headroom for other devices and always granting at least one. virtio-blk uses it to cap its active queue count; extra probed virtqueues are simply left unused.

Verification

  • Applies clean on master; all 5 changed files compile clean; the whole kernel compiles (1198 objects) with the patch applied.
  • reserve_msix_vectors resolves within the driver files (defined in virtio.o, referenced in virtio-blk.o); a partial link of the changed objects resolves with no dangling references.
  • The boot A/B (a high-vCPU guest with several multiqueue virtio-blk devices aborting on stock, booting on this change) was demonstrated during the work this was extracted from; this standalone extraction is compile- and link-verified.

The x86-64 IDT exposes only 224 usable interrupt vectors (32..255), and
every device shares that pool.  A guest with many vCPUs and several
multiqueue virtio devices can request one MSI-X vector per queue and
exhaust the pool during boot.  Two things then went wrong:

  - register_interrupt_handler() called abort() when no IDT vector was
    free, crashing the boot instead of letting the device degrade.
  - request_vectors() pushed a msix_vector whose allocation had failed,
    and ~msix_vector() then unregistered vector 0.

Fix both and add a soft cap so devices stay within budget in the first
place:

  - register_interrupt_handler() returns 0 (an impossible real vector,
    since 0..31 are CPU exceptions) as an "exhausted" sentinel instead
    of aborting.  msix_vector's destructor skips the unregister when its
    vector is 0, and request_vectors() stops at the first failed
    allocation and returns the short vector it did get; the existing
    easy_register() already treats a short result as failure, so a
    driver falls back to fewer queues rather than crashing.
  - Add virtio_driver::reserve_msix_vectors(): a process-wide soft
    budget that a multiqueue driver consults to cap how many queues it
    arms with a distinct interrupt, leaving headroom for other devices
    and always granting at least one.  virtio-blk uses it to cap its
    active queue count; extra probed virtqueues are simply left unused.

Signed-off-by: Greg Burd <greg@burd.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant